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ABSTMCT 



This study examined the impact of achievement testing on a Spring- 
Spring vs, a Fall^Spring basis in avaluation of compensatory education 
programs. The affects of the summer vacation period and crossing test 
levels were examined.. Significant decreases in achievemant test scores 
were founds at all grade levels , between Spring and Fall test adminis-- 
trations* Substantially larger decreases were noted when testing levels 
were crossed. 

The implications of differences in achievement scores s a function 
of time and level of testing, are discussed in terms of the KMC Title I 
evaluation models and the current emphasis placed upon longitudinal 
examination of compensatory programs* 
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Title I of the Elementary and Secondary Education Act of 1965^ as 
amended, (ESEA) authoriges financial assistance to local school districts 
with concentrations of economically disadvantaged students* Thasa 
fumls are designaced specifically for programs designed to meat the 
special needs of the educationally disadvantaged student. Tha lagisla- 
tion for Title I, (Public Law 93-380, section 151) , requires that the 
Commissioner of Education '"provide for Independent evaluations which 
describe and messure the Impact of programs and projects assisted under 
this title-" In order to provide a uniform and. standard format of 
reporting impact^ the United States Office of Education (USOE) has 
awarded two contracts to BMC Research Corporation for the devalopmant 
and refinement of a series of models for evaluating the coginitlve 
outcomes of Title I compensatory education academic programs* 

In docuraents explaining the evaluation models developed under 
these contracts RI^IC Research Corporation has identified sevaral 
'^hazards" which, if violated^ would invalidate the evaluation findings* 
One hazard cited, the use of non-comparable (non^-ldenttqal) pretest and 
post test levels, becomes a severe restriction upon the avaluation of 
local Title I programs when annual (Spring to Spring) testing is used 
for assessing achievement* Since Spring to Spring testing is CQmmon In 
many school districts, the problem of crossing testing levels becomes 
acute when exatnined in light of this hazard. If this limitation is 
a valid one then the probletn becomes even more acute and cachnlques 
developed by R>1C for aggrGgatlng data acroas school district and 
states are employed. . . 

A further problem associated with Spring to Spring testing is the 



hypothesis of che "suimner drop*^ Since many communities examine progTam 
Impact on a Fall--Spring basiBj the effect of this Bumsner drop will make 
comparison (or aggregation) of scores froin commuriities which test on a 
Fall'-Spring schedule with those following a Spring-Spring testing 
schedule eKCremely difficult if the magnitude of the summer drop Is not 
taken Into account. 

METHOD 

The sample consisted of 238 scudents classified as educationally 
disadvantaged in grades two through six in a moderate-^alEed BJiode Island 
school district* One hundred twenty-six (126) students smre tested with 
the California Achievement Test (CAT) at two testing tines with the same 
level instruments while one hundred twelve (112) students were given a 
different level of the test for the second administration* Table 1 pre- 
sents numbers of students ^ test daces ^ levels and forms for the sample in 
the study. 



TABLE 1 



Test Levels and ForTOS by Grade 





Sprlnft, 1974 


Fall, 1975 


Grade 2 
N ^ 69 


Level 1, Form B 


Level 2j Form A 


Grade 3 
N ^ 60 


Level 2, Form B 


Level 2, Form A 


Grade 4 
N - 43 


Level 2^ Form B 


Level 3j Form A 


Grade 5 
N ^ 66 


Level 3 ^ _Form _B _ 


Level 3, Form A 
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RESULTS . 
Tabic 2 presents, for each of the four grades eKamlned as part of 
the study, the Spring, 1974 and Fall^ 1975 mean Achievement Devalopment 
Scale Scores (ADSS) * Examination of these scores indicates that there 
was a decline In mean ADSS at all grade levels between Fall and Spring 
test adtrilniscratlons. At 3 of the 4 grade levels ^ these declines were 
statistically significant at the .01 level. 

TABLE 2 ' f 

Mean ADSSy Standard Deviation^ t Values and Significance for 
California Achieyemejit Test Total Reading 





Spring, 1974 


Fall, 1975 


t 


P 




SD 




SD 


Grade 2 
CN=69) 


309 


21 


281 


29 


8.12 


<. 01 


Grade 3 

(Na60) 


312 


30 


305 


31 


1.78 


M.S. 


Grade h 
(N-43) 


347 


34 


336 


31 


2.82 


<.01 


Grade 5 
CN=66) 


362 


35 


344 


37 


4.48 


<.01 



One of the key elements in the R>iC Research Corporation evaluation 
models is the emphasis upon a clear definition of "no--treatment expectation' 
i.e.yihow well the students t^ould be predicted to achieve without supple^ 
mentary assistance. Since in Model A, the norm-'referenced models the 
no^-treatment eKpectation is based upon percentile rank associated with 



mnan prLtLfjBt standard scar© ^ mean standard scores for the testing in 
this study were convert ad to percentiles* Table 3 presents these 
Derce!! tiles * 

TABLE 3 

Percent i l^ Rank, Both Administrations for 
Calif ornia_Achlevernent Test^Qtal Reading 





Spring, 1974 
Percentile Rank 


Fall, 1975 
Percentile Rank 


Grade 2 


33 


12 


nradd 3 


14 


10 


Grada 4 


20 


11 


Grade 5 


13 


8 



The examination of these percentiles highlight several points. 
Firsts in all cases^ Fall percentile scores are lower than Spring percen- 
tile scores. Second J where levels of the CAT have been crossed (grade 
two and four), Sprlng^Fall declines in percentile scores are greater 
than where cor.mion test levels are used (grade 3 and 5)* Thirds Fall 
percentile scores are more homogenous across grade levels than are 
percentile scores for Spring testing. 
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DISCUSSION 

The proceding section has shown that achievement test scores earned 
by the same sCudenCs are different when these students are tested ii^ the 
Spring and the Fall* As a result of these differences^ the use of studants 
Spring achievement test scores as pretest measures will yield different 
predicted postteat scores than If their Fall achievement test scores 
were usad as pretest measures* In all cases, predicted posttest perf ord- 
inance will be greater If Spring scores are used^ — hence making It less 
likely that assessment of program impact will identify "significant 
Improvement" if Spring scores ate used as the pretest (basis for predic- 
ting expected posttest performance)* This phenomenon is particularly 
noticeable where test levels change from Spring to Fall administration. 

Not only does Spring vs. Fall pretest administration appear to 
Impact f^pon whether or not an individual program Is identified as having 
"signif icant"irjpact upon student achieveinent ^ but there also appears to 
be serious implications of Spring-Sprltig versus Fall-Spring testing 
models upon comparisons or aggregationi of scores obtained via these 
different testing models. At all grade levels a program using Fall- 
Spring testing will yield a more positive growth factor— whether measured 
in ADSS, percentile or NCE units-«-than will the same proirara using Spring- 
Spring testing. Attempts to compare ot aggregate across programB, dis- 
tricts or states^ must, therefore, be cognizant of the particular testing 
modal followed. Comparisons or aggregations across testing models would 
appear to be Inappropriate. 

i 

IMPLICATIONS 

This investigation provided empirical data related to several edu- 
cationiil questions. As mentioned earlier, the EMC Title I evaluation 
models rcquirG that the same level of the test be administered for pro- 



and posttasting to miniraize errors. This requirement has been criticiEed 
by inany who would like to cross tosting levels as students change grade 
levels. Since greater Spring-Fall decreases were found when test levels 
were crossed , this Inves cigat ion tends to support this WLC model require- 
ment* 

In addition, this study provides an estimate of the effects of the 
"summer drop," It is a widely accepted but little researched belief 
that students lose some of the growth over the summer that they have 
achieved during the school year. This study provides some empirical 
evidenc^j with respect to reading achievement , to support this belief. 
The logical Implication of this finding would be that a longitudinal 
examination of student achievement growth over long periods of time may 
well show that "the whole is less than the sum of its parts i.e* student 
growth across a period of years may well be substantially below the 
cumulative total of yearly achievement gains measured by Fall^Sprlng 
achievement testing. 
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